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INTRODUCTION 


The U.S. is in the midst of a college completion crisis. While high school graduation rates have risen for 

two decades to a record high of 86 percent,' college graduation rates have stagnated and rates for students 
of color have remained below even that stagnate rate.? The growing cost of college is one, albeit perhaps 

the most significant, barrier to completion students face. However, research shows that frequent, intensive 
advising combined with financial and other support can make a tremendous difference in helping students 
complete.*That is why two years ago, TICAS began working with a set of organizations across the country that 
are providing these comprehensive approaches to student success, or CASS. 


These organizations, which include CUNY ASAP, InsideTrack, MAAPS, One Million Degrees, Project 

QUEST, Stay the Course, and Bottom Line all take a similar approach of connecting students with a 

counselor or case manager who meets frequently with them to build a personal relationship and connect them 
with a customized suite of support - academic, financial, and personal - to help them overcome barriers and 
attain a degree.‘ 


All of these programs have been rigorously evaluated and found to measurably increase college persistence, 
completion, and/or student earnings upon graduation. CUNY ASAP, the most well-known of these initia- 
tives, doubled college graduation rates among participants. For those students who started working with One 
Million Degrees in high school, graduation rates increased by 73 percent.° In the 11 years following their exit 
from ProjectQUEST, participants continue to earn more than individuals who did not participate, about 
$31,000 more.* Yet, despite the clear promise they show, these programs have struggled to secure reliable, 
sustained public investments that would make it possible for them to scale nationally and move the needle on 
college completion rates across the country. 


TICAS convened these organizations quarterly, along with research partners MDRC and the Wilson 

Sheehan Lab for Economic Opportunities (LEO), to better understand what it would take to reach all students 
who could benefit from these programs throughout the country. Through this “community of practice,” TICAS 
also sought to provide a forum for leaders of these organizations to share learnings and challenges with one 
another to accelerate the growth of the field. 


nitially, TICAS sought input from all members of the community of practice to craft a national policy 
proposal aimed at providing federal funds to states to make CASS programming more widely available.’ 
President Biden and members of the U.S. Congress included components of this proposal in their plans 

to provide significant new investments in evidence-based college completion programs, $62 billion over 

10 years in the American Families Plan and $9 billion over 7 years in the Build Back Better legislation. The 
Build Back Better bill has passed the House of Representatives and is now being considered by the Senate. 
t may result in significant new resources for these evidence-based interventions. If so, policymakers 

and program and institutional leaders will immediately wrestle with critical questions about how to scale 
with fidelity, particularly in different geographic locations and institutional contexts, and strengthen the 
field's impact on equity. To crystalize the elements of those key questions the field needs to answer, TICAS 
engaged community of practice members in creating a research agenda and equity framework. This report 
summarizes those efforts, including the research questions the community of practice has surfaced as most 
pertinent to prioritize and a framework for equity in the use of data that each organization is interested in 
adopting. 


RESEARCH AGENDA 
Existing Studies 


Multiple randomized controlled trials (RCT) show that CASS programs are effective at improving stu- 
dent outcomes, ranging from college persistence and credit accumulation to longer-term impacts on 
completion, transfer, and earnings. Indeed, evidence on CASS programs represents the strongest body 
of evidence about what works to improve postsecondary education. The findings from the RCTs of the 
programs involved in our community of practice provide a snapshot of the robustness of the 

CASS evidence base: 


CUNY ASAP: Through both longitudinal program data and external RCT evalua- 

tion, the graduation rate of ASAP participants, virtually all of whom are low-income stu- 
dents of color, has been shown to be nearly double that of their comparison group.’ The 
evaluation also concluded that the intervention held promise for narrowing equity 

gaps. Additionally, the program saw similar gains in student completion rates at three com- 
munity college replication sites in Ohio.'? A study of ASAP's impact after six years found 
that program participants continued to graduate at higher rates than other students. 


Project QUEST: An external RCT evaluation with additional state administrative data found 
that QUEST participants earned over $4,600 more annually in the eleventh year after 
exiting the program and more than $31,000 over the entire 11-year follow-up period, 
compared to those in the control group. While previous studies on Project QUEST have 
shown gains in short-term earnings among participants, overwhelmingly students of color 
who are living in poverty, this long-term study result shows that the program has significant, 
positive impacts on participants’ career development." 


One Million Degrees (OMD): In an external RCT evaluation across eight community college 
campuses in Chicago, initial findings show that OMD’'s model increases associate's degree 
attainment by 19 percent for students who took up the offer to participate in the program 
and enrolled in college compared to the control group.’* For students who begin with OMD 
in high school, the study found substantially larger impacts. Close to 95 percent of students 
served by OMD are non-white and almost 70 percent are eligible for Pell grants, positioning 
the program well to close equity gaps in college graduation rates. 


InsideTrack: An external RCT conducted using InsideTrack data from eight different 
postsecondary institutions in the 2003-2004 and 2007-2008 school years found that 
students who were coached in the first year of entering the program were about 15 percent 
more likely to persist in college after 18 and 24 months of receiving coaching.® 


Stay the Course: An RCT evaluation at a large community college in Texas found 

that Stay the Course increased associate's degree completion among enrolled women by 
31.5 percentage points, which is nearly three times the graduation rate of women in the 
control group. Among program participants, the average family income was just above 
$22,500, 90 percent lived at or below 200 percent of the federal poverty limit, and 66 per- 
cent were Black or Hispanic. 


Bottom Line: A large, multi-site RCT evaluation with administrative and survey 

data found Bottom Line's program model - intensive advising during high school and col- 
lege — increased the likelihood of participants enrolling in four-year colleges by 13 per- 
cent. The study also found that participants were likelier to enroll in higher quality insti- 
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tutions compared to the control group. Bottom Line students were also 53 percent more 
likely to earn a bachelor’s degree within six years after high school."® Students are eligible for 
Bottom Line if their families make less than 200 percent of the federal poverty limit. 


e Monitoring Advising Analytics to Promote Success (MAAPS): An external RCT study 
with an over 10,000-student sample from 11 institutions found that after four academic 
years, students at Georgia State University who were offered MAAPS advisement had a 
GPA that was 0.16 higher than the control group; student subgroups such as those who are 
Pell-eligible, first-generation, and underrepresented saw similar gains. Black students in the 
treatment group also had higher graduation and persistence rates that were eight and 12 
percentage points higher than their counterparts in the control group, respectively." 


Expanding and replicating evidence-based models such as these will be critical to closing educational 
equity gaps nationwide. These programs all primarily serve students from low-income backgrounds and 
students of color, and they are designed to close equity gaps. National higher education researchers 
and experts suggest that offering students both academic and non-academic supports, such as 
proactive advising, career guidance, and financial aid, can help them navigate a variety of challenges 
that impede them from completing coursework and earning a postsecondary degree. Specitical- 

ly, they point out - among other strategies for providing comprehensive student supports at the com- 
munity college level - the importance of connecting with students one-on-one to assess their needs 
and challenges.” 


Proposed Research Agenda 


Although a few programs such as CUNY ASAP and InsideTrack have been able to demonstrate 

the effectiveness of their program at replication sites, the robustness of other replication efforts has 

yet to be determined.” Problems finding enough students at replication sites to take up the program, 
expansion sites that deviate from the core components of the program, and fickle funding streams have 
all created hurdles for organizations attempting to replicate and scale their original program with fidelity 
in other states and institutions. CUNY ASAP. which has been able to overcome these hurdles success- 
fully for the most part, has done so by dedicating staff to providing technical assistance to its replication 
sites full-time and limiting the number of new institutions it will assist in any given year. 


The promise these proven CASS programs have shown in increasing college completion, particularly 
for students that have traditionally been under-served, is simply too great not to have them operating 
successfully in more places across the country. To do so, however, policymakers and program leaders 
need to understand better how to replicate and scale without losing the components that have made 
them successful. Thus, more research is needed on how proven programs can successfully replicate 
their models in different contexts while maintaining the same degree of impact they experienced at 
their original location. Over the course of four half-day convenings, the programs in our community of 
practice discussed the need for further evidence on this issue as well as other areas where the evidence 
base is lighter. They concluded that the two highest priority areas of needed research are (1) replication 
with fidelity and (2) long-term impacts. 


To differing degrees, the programs within the community of practice all are wrestling with questions 
about how to expand effectively while adhering to the core components of their model and adjusting 
and adapting other elements to provide the greatest benefit to students. Centrally, program leads are 
wondering what outcomes and other data elements to measure to track program quality as they ex- 
pand. 
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The second major bucket of research program leads coalesced around during community of practice 
meetings was measuring the long-term impact of program participation on students’ economic mobil- 
ity. The community of practice members have varying degrees of evidence about the impact of their 
programs on short-term metrics like persistence, completion and even initial student earnings, but they 
lack long-term evidence of the implications of participating in their program for students’ economic 
mobility. (Project Quest is the one exception, with documented causal impact on long-term earn- 

ings.) Partly programs lack long-term impact data because they have not been operating long enough 
to have developed that kind of evidence. But data limitations also contribute to the problem. Although 
most states have a longitudinal data system, few link their postsecondary education system with their 
wage data (generally through their unemployment insurance system) consistently. And, because CASS 
programs are external nonprofit providers and not state-run higher education institutions (except for 
CUNY ASAP) the difficulty obtaining data about their students’ post-college lives currently is very diffi- 
cult. But, as with the replication research, for students to benefit fully from these programs more needs 
to be known about their impact on long-term economic mobility. 


Drawing from these two areas of research interest, community of practice leads articulated these 
two specific research questions and sub-questions: 


RESEARCH QUESTION #1 
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programs scale up within institutions, across states, and across the nation? 


e How have program modifications affected student success outcomes, such as persistence 
and completion? 


e What types of evidence might program and institutional leaders rely on to measure these 
modifications and their impacts? Is there a way to use qualitative data? 


e What are the best practices shared across all CASS programs (or that could be shared) on 
modifying with fidelity as a program expands? 


e Are there tools that could be created, for instance, such as benchmarking templates or data 
reporting requirements, to better track quality while scaling? 


e How should programs measure fidelity when some parts of programs are replicated identi- 
cally to the existing program and other parts are modified? 


e How should program leaders and researchers weigh various components and modifications 
to determine the level of fidelity? 


e How can stakeholders determine whether a variation is significant enough to merit additional 
rigorous evaluation to ensure positive impacts? 


e What are the implications and considerations for replication with fidelity based on this 
evidence? 


e Are there program elements that are more or less critical to maintain fidelity? 


e What do these answers suggest about the cost of replication? 
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RESEARCH QUESTION #2 


Do these programs result in increases in students’ earnings, income, wealth, and 


well-being over the long-term? Do these programs enable students to achieve 
family-sustaining jobs throughout their lives? 


e What metrics should students use to track students beyond graduation (ideally, 3- or 4-year 
outcomes or longer)? Are there outcomes beyond persistence, credit accumulation, receipt 
of a degree or certificate, and earnings upon graduation? 


e ls it possible to collect and disaggregate data by demographic groups beyond graduation? 
What are the barriers to such an approach and how could they be overcome? 


e Could researchers and program leads extend their follow-up research and data collec- 
tion periods to include transfer rates, bachelor’s graduation rates, and earnings, occupation 
or employment information at the 6- to 8-year mark? 


e Could states facilitate these data collection approaches? 


e Could programs define a common set of outcomes, such as persistence, transfer, receipt of 
a 4-year degree, including disaggregation by student subgroup, that all programs, institutions 
and states collect and report? 


e ls there a way to look at the impact of these programs over an even longer timeframe? 


Proposed Equity Framework 


In addition to the above research questions, the community of practice coalesced around a shared 
equity framework, which will serve two purposes. First, this framework allows programs to measure and 
begin to close internal equity gaps. Second, the framework provides a uniform set of metrics to help 
programs describe their impact to external audiences. 


A commitment to equity is critical to ensuring that underserved students benefit from these services. 

If programs take a passive approach to enrolling students, they could end up serving students who are 
more motivated to succeed than others, or who share characteristics that make them more likely to 
persist in and graduate from college, regardless of whether they participate in a CASS program. While 
the evaluations did not find substantial demographic differences between the treatment and control 
groups, program leads are motivated to closely and regularly monitor data about program participants 
to ensure that they are using their limited resources to serve students who are low-income, BIPOC, and 
otherwise have traditionally been underserved by educational systems. Without a commitment to equi- 
ty, these programs could exacerbate racial gaps in college completion. 


The foundation of the shared equity framework for CASS programs is disaggregated student participant 
and outcome data collected at the same times throughout students’ educational journey. Specifically, 
programs plan to collect demographic data at entry, compare the population of participants to the in- 
stitutional demographics and work to ensure the program is reaching all students, including the hardest 
to reach students. At minimum, programs will work to serve more Black, Indigenous, Latinx, and other 
people of color as well as first generation students than the student population writ large. 
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Programs in the community of practice operate in different contexts. Project QUEST, for example, 
serves mostly Latinx students while One Million Degrees and CUNY ASAP serve more Black stu- 
dents. All three programs, however, serve a large population of first-generation students. While the 
demographic profiles of the programs, and the institutions and states in which they operate vary, the 
programs are committed to defining a common set of demographics of interest and measuring out- 
comes for each group to ensure the programs are promoting equity internally and narrowing equity 
gaps across their institutions. 


Some of the key variables included in the equity framework are: 


BASELINE VARIABLES | OUTCOME VARIABLES 
Registration 
Demographics Enrolled at 4-year 
Race/Ethnicity Total credits attempted 


Credits Attempted 
Children 


Plan to work during the Attained an associate's 
semester degree 


Credits Attained 
Currently employed College credits earned 
Average weekly hours Attained a certificate 
Employment 
Credentials Attained 
a3 Attained a bachelor’s 
Pell recipient 
; Berrie degree 
Financial Aid 


Expected Family Attained a graduate 
Contribution degree 


Program leads are committed to using this data to improve their practices, including how they structure 
their programs and what kinds of supports they provide to students, and the practices of the institu- 
tions within which they work to advance equity. Fundamentally, the programs want to drive systematic 
changes in how their institutions serve first-generation students, students of color, and students from 
low-income backgrounds to college completion. They raised the following specific questions in consid- 
ering which data to collect and how to think about using it in their practices: 


e Why are we collecting this data and how will we use them to change programmatic and insti- 
tutional practices to better support students of color, first generation students and students 
from low-income backgrounds? 


e How do we collect data while respecting the dignity of students to only share what programs 
need to support them and understand program impacts? 


e How do we measure equity gaps (one static goal/benchmark, reduction in the gap)? 


e How do we define subgroups of students, including accounting for some intersectionality? 


e What effects are community of practice programs having on broader equity gaps within their 


institutions? 
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e What are the equity gaps within community of practice programs? 


e Are there models or examples of robust equity frameworks we could draw from? Catholic 
Charities of Fort Worth, for example, has a set of guiding principles aligned to their data eq- 
uity framework that helps program leaders prioritize inclusivity and equitable data collection 
when considering what data they need to collect to suit their needs. 


Where do we go from here? 


Over the course of the next two years, TICAS will work with our research partners, MDRC and 

LEO, to begin answering these questions and to share the findings with federal and state policymakers 
and the student success field. The community of practice will begin implementing the shared equity 
framework this year. We invite other programs to establish their own equity frameworks, borrowing from 
ours as a basis. We also invite other researchers to work with us to answer these questions and help 
contribute to the body of knowledge that is leading the way for helping underserved students’ complete 
college and advance equity in postsecondary attainment. 


ENDNOTES 


1 U.S. Department of Education, National Center for Education Statistics. May 2021. “Public High School Graduation Rates.” 
https://bit.ly/3HH8EFV. 

2 U.S. Department of Education, National Center for Education Statistics. “Graduation Rate Within 150% of Normal Time at 2-Year Postsecondary 
Institutions.” https://bit.ly/3mZJ6mv; U.S. Department of Education, National Center for Education Statistics. “Graduation Rate Within 150% of 
Normal Time at 4-Year Postsecondary Institutions.” https://bit.ly/3tl6gNh; U.S. Department of Education, National Center for Education Statistics. 
February 2019. Status and Trends in the Education of Racial and Ethnic Groups. https://bit.ly/3G69lzm. 

3 Alexander Mayer and Kate Tromble. November 2020. Understanding What Works in Postsecondary Education. MDRC. https://bit.ly/34qAFdy. 

4 TICAS. July 2021. Comprehensive Approaches to Student Success Design Principles. https://bit.ly/3G2Rn0C. 

5 Kelly Hallberg et. al. November 1, 2021. Supporting Community College Student Success: Evidence from a Randomized Controlled Trial. Univer- 
sity of Chicago Urban Labs. https://bit.ly/3dcRzhO. 

& Anne Roder and Mark Elliot. September 2021. Eleven Year Gains: Project QUEST'’s Investment Continues to Pay Dividends. Economic Mobility 
Corporation. https://bit.ly/3ojfhhP. 

7 TICAS. February 2021. “Scaling Student Success: A National Proposal.” https://bit.ly/3G12JSK. 

8 U.S. White House. April 28, 2021. “Fact Sheet: The American Families Plan.” https://bit.ly/3FV4mBg; U.S. House of Representatives. September 
2021. Repository for the Committee on Education and Labor. https://bit.ly/3n12BvO. 

° CUNY. October 2021. Significant Increases in Associate Degree Graduation Rates: CUNY Accelerated Study in Associate Programs (ASAP). 
https://bit.ly/2ZHO9kM. 

Cynthia Miller et. al. January 2020. Increasing Community College Graduation Rates with a Proven Model. MDRC. https://bit.ly/3phHsgl. 

" Anne Roder and Mark Elliot. September 2021. Eleven Year Gains: Project QUEST’ Investment Continues to Pay Dividends. Economic Mobility 
Corporation. https://bit.ly/3ojfhhP. 

® Kelly Hallberg et. al. November 1, 2021. Supporting Community College Student Success: Evidence from a Randomized Controlled Trial. Univer- 
sity of Chicago Urban Labs. https://bit.ly/3dcRzhO. 

® Eric P. Bettinger and Rachel B. Baker. September 18, 2013. The Effects of Student Coaching: An Evaluation of a Randomized Experiment in Stu- 
dent Advising. Educational Evaluation and Policy Analysis. https://bit.ly/3xMuds4. DOI: 10.3102/0162373713500523. 

4 William N. Evans et. al. December 2017. Increasing Community College Completion Rates Among Low-Income Students: Evidence from a Ran- 
domized Controlled Trial Evaluation of a Case Management Intervention. https://bit.ly/2ZL2kDX. 

® Andrew Barr and Benjamin Castleman. October 2021. The Bottom Line on College Advising: Large Increases in Degree Attainment. 
https://bit.ly/3ofBNbw. 

% Daniel Rossman et. al. June 24, 2021. MAAPS Advising Experiment. ITHAKA S+R. https://bit.ly/31tPHNW; Georgia State University was the only 
institution out of the 11 that offered MAAPS advisement beyond the first three years of the intervention and was the only institution that saw statis- 
tically significant findings. 

” Hope Cotner et. al. 2021. Designing and Delivering Career Pathways at Community Colleges: A Practice Guide for Educators (WWC 2021007). 
National Center for Education Evaluation and Regional Assistance (NCEE), Institute of Education Sciences, U.S. Department of Education. 
https://bit.ly/3De9ApL. 

® Lab for Economic Opportunities. August 2020. Comprehensive Approaches to Increasing Student Completion in Higher Education: A Survey of 
the Landscape. https://bit.ly/30fZPTq. 


The Institute for College Access & Success Page 7 


ACKNOWLEDGMENTS 


The Institute for College Access & Success (TICAS) is a trusted source of research, design, and advocacy for student-cen- 
tered public policies that promote affordability, accountability, and equity in higher education. To learn more about TICAS, 
visit ticas.org and follow us on Twitter and Instagram: @TICAS org. 


This report was co-authored by Ana Fung, Catherine Brown, and Kate Tromble. 


TICAS received generous support for this project from Arnold Ventures, the Bill & Melinda Gates Foundation, Joyce 
Foundation, and Robin Hood Foundation. The views expressed in this paper are solely those of TICAS and do not neces- 
sarily reflect the views of our funders. 


This report can be reproduced, with attribution, within the terms of this Creative Commons license: creativecommons. 
org/licenses/by-nc-nd/3.0/. 


the institute for 
college 
access & success www.ticas.org 


110 Maryland Ave, NE, Suite 201 
Washington, DC 20002 
202.223.6060 


